A Conceptual Study of Model Selection in Classification - Multiple Local Models vs One Global Model

A key concept in model selection is to understand how model complexity can be modified to improve in generalization performance. One design alternative is to increase model complexity on a single global model (by increasing the degree of a polynomial function); another alternative is to combine multiple local models into a composite model. We provide a conceptual study that compares these two alternatives. Following the Structural Risk Minimization framework, we derive bounds for the maximum number of local models or folds below which the composite model remains at an advantage with respect to the single global model. Our results can be instrumental in the design of learning algorithms displaying better control over model complexity.